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Executive  Summary 


This  report  documents  the  proceedings  of  the  Automated  Author  Aiding 
Systems  Conference  held  at  UCLA  on  June  6-7,  1984.  The  purpose  of  the 
conference  was  to  determine  the  current  state  of  the  art  in  computerized 
aids  for  authors  by  providing  a  forum  for  the  exchange  and  synthesis  of 
ideas,  as  well  as  the  demonstration  of  extanr  system*.  Individuals  invited 
to  the  conference  included  researchers  from  tnc  ■  partrnent  oi  Defense  (Dob) 
and  the  academic  sector  with  interests  in  the  area,  representatives  ot  UoD 
agencies  and  other  organizations  with  interests  in  the  development  and 
application  of  such  systems,  and  representatives  of  vendors  of  author 
aiding  systems. 

The  conference  highlighted  issues  which  must  be  addressed  in  the 
further  developments  of  author  aiding  systems.  These  include: 

1.  The  technical  adequacy  of  ISP.  While  specific  concerns  were 
raised  about  the  ISD  model,  there  was  general  agreement  that  some  type  of 
logical  analysis  and  development  procedure  such  as  ISD  is  necessary  to 
design  any  instruction.  In  fact,  there  is  some  evidence  that  applying  ISD 
procedures  to  courses  before  they  are  presented  on  computers  may  actually 
account  for  their  success  rather  than  the  computer  technology  itself  (see, 
for  example,  the  Orlansky  and  String's  review  of  computer  based  instruction 
in  the  military).  While  there  is  evidence  to  believe  that  ISD  has  value  as 
a  technical  approach,  certain  limitations  and  technical  difficulties  were 
consistently  noted  by  conference  participants.  For  example,  while  the 
procedural  steps  in  ISD  are  logically  organized,  some  of  the  blocks  within 
the  ISD  process  require  complex  and  difficult  decisions,  particularly  the 
initial  blocks  of  front-end  analysis  activities,  involving  task  analysis 
and  instructional  objectives  definition.  These  analysis  activities  are 
often  not  straightforward,  particularly  for  complex  cognitive  job  tasks; 
further  complicating  the  analytic  work  is  this  practical  circumstance  that 
access  to  subject  matter  experts  is  usually  quite  limited.  Furthermore,  it 
is  often  hard  for  subject  matter  experts  to  articulate  the  key  features  of 

a  task  or  to  describe  standards  for  determining  the  adequacy  of  task 
performance.  This  is  especially  true  for  cognitive  tasks  requiring  higher 
order  processing  of  information  and  decisioi  -making.  Greater  attention 
needs  to  be  directed  toward  articulation  oi  how  to  perform  ISD  procedures 
more  effectively. 

2.  The  Implementation  of  ISD.  One  thread  that  ran  throughout  the 
comments  of  many  participants  was  the  administrative  and  management 
problems  associated  with  the  implementation  of  ISD  which  have  impeded  its 
success.  These  problems  include  Inadequate  training  of  ISD  manayers  and 
practitioners  as  well  as  inadequate  resources  (including  time  and 
expertise)  applied  to  ISD  procedures.  In  addition,  communication  with  job 
managers  in  the  field,  as  well  as  scheduling  constraints  and  availability 
of  subject  matter  experts  contribute  further  to  problems  in  the  effective 
implementation  of  the  ISD  model. 


3.  Authoring  by  individuals  with  team  review..  While  individuals 
typical ly' write  initial  drafts  (e.g.,  a  task  analysis,  training  objectives, 
lesson  plans,  a  test),  their  work  is  generally  reviewed  extensively  both  by 
other  individuals  and  by  committees.  The  consensus  view  appeared  to  be 
that,  not  only  would  intelligent  computer  aiding  contribute  to  the 
productivity  and  effectiveness  of  individual  authors,  but  that  the  review 
process  could  be  made  shorter  and  more  efficient  with  an  aiding  system  that 
incorporated  such  features  as  E  mail  and  word  processing. 

4.  The  role  of  cognitive  science.  There  is  a  great  deal  of  talk 
about  the  importance  of  incorporating  principles  of  cognitive  science  into 
the  1SD  process  or  of  basing  our  notions  about  instructional  development  on 
ideas  emerging  from  cognitive  science  and  the  study  of  human  Information 
processing.  However,  it  is  very  difficult  for  instructional  developers  and 
users  of  ISO  to  articulate  exactly  how  specific  principles  from  cognitive 
science  were  or  could  be  incorporated  into  their  instructional  development 
activities.  More  attention  needs  to  devoted  to  this  issue. 

5.  The  range  of  training  problems.  Sources  of  training  problems  not 
only  Include  problems  stemming  from  the  ISO  process  itself,  but  also 
include  problems  arising  from  a  broad  range  of  other  sources  specifically 
including  selection  and  placement  problems  (e.g.,  trainees  with  inadequate 
entering  skills  and  knowledge),  resource  allocation  problems  (e.g., 
insufficient  time  for  accomplishing  training  objectives),  and  nanagement 
problems  within  the  employing  organization  rather  than  the  training 
organization  (e.g.,  time  lag  between  training  and  job  assignment  resulting 
in  skill  decay,  mismatch  between  training  and  job  requrlements).  Effort 
should  be  directed  to  examining  the  nature  of  "training'1  problems  beyond 
the  context  of  ISO  narrowly  conceived  and  alternative  solution  strategies 
considered. 

6.  The  need  for  a  systems  approach.  There  is  a  need  for  ISO  to  be 
incorporated  into a  broader  systems  view  so  that  ISD  does  not  start  with 
the  assumption  that  training  is  the  appropriate  solution  to  a  given  problem 
or,  equivalently,  so  that  ISO  does  not  stop  with  training.  A  true  systems 
approach  would  consider  the  full  range  of  personnel  and  organizational 
management,  not  just  the  contributions  of  training  and  would  provide  a 
mechanism  for  helping  to  decide  if  the  cost-effective  solution  is  to  seek 
better  training.  In  a  given  situation  it  might  be,  for  example,  that  the 
cost-effective  approach  is  to  seek  better  personnel  selection,  or  better 
classification,  or  better  job  design,  or  better  equipment  design,  or  better 
job  aids,  or  better  supply  schedules,  etc.  Thus,  ISD  ought  to  be  viewed  in 
a  context  broader  than  training. 

7.  The  capabilities  of  extant  hardware  and  software.  There  is  a 
common  perception  that  efforts  in  training  lag  behind  the  computer 
revolution  in  software  and  hardware.  However,  there  are  ISD  needs  that  far 
outstrip  the  capabilities  of  current  technology.  This  situation  is 
particularly  true  in  the  field  of  artificial  intelligence  and  has  direct 
implications  for  ttie  development  of  an  appropriately  powerful  A1  system  for 
ISD. 
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8.  The  role  of  artificial  intelligence.  While  artificial 
intelligence  (AT)  is  often  viewed  as  having  great  potential  for  application 
to  the  ISO  process,  there  may  be  limitations.  If  ISD  is  conceived  as 
common  sense  made  systematic,  then  it  may  be  anomalous  to  consider  Al  as  a 
promising  approach  in  ISD  since  the  common  lament  among  AI  workers  is  their 
inability  to  capture  "common  sense".  It  is  also  important  to  consider  the 
costs  of  adding  AI  to  the  ISD  process.  That  is,  it  will  be  important  to 
estimate  whether  the  costs  incurred  are  compensated  for  by  increased 
instructional  effectiveness  or  increased  productivity  among  instructional 
developers. 

In  addition,  the  conference  identified  foci  which  should  guide  the 
development  of  automated  author  aiding  tools.  These  were: 

o  a  focus  on  individual  worker  rather  than  group  training; 

o  use  of  a  technical  "hard"  skill  domain  such  as  operator  or 
maintainance  training  rather  than  a  "soft"  skill  domain 
such  as  leadership; 

o  requirement  for  adequate  rather  than  optimal  Instruction; 

o  in-school  instruction  rather  than  unit  training; 

o  off-the-shelf  hardware  rather  than  hardware  development; 

o  examination  of  existing  systems  and  identification  of  needed 
enhancements; 

o  consideration  of  artificial  intelligence  techniques  particularly 
for  the  knowledge  base  of  instructional  principles; 

o  rapid  development  with  formative  evaluation  and  revision  rather 
than  extended  development  cycle. 
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AUTOMATED  AUTHOR  AIDING  SYSTEMS  CONFERENCE 
Final  Report 

Introduction 

This  report  documents  the  proceedings  of  the  Automated  Author  Aiding 
Systems  Conference  held  at  UCLA  on  June  6-7,  1984.  The  purpose  of  the 
conference  was  to  determine  the  current  state  of  the  art  In  computerized 
aids  for  authors  by  providing  a  forum  for  the  exchange  and  synthesis  of 
Ideas,  as  well  as  the  demonstration  of  extant  systems.  Individuals  Invited 
to  the  conference  Included  researchers  from  the  Department  of  Defense  (DoD) 
and  the  academic  sector  with  Interests  in  the  area,  representatives  of  DoD 
agencies  and  other  organizations  with  Interests  In  the  development  and 
application  of  such  systems,  and  representatives  of  vendors  of  author 
aiding  systems. 

Approximately  75  Individuals  attended  the  conference.  (A  list  of 
participants  Is  included  In  the  Appendix  to  this  report.)  The  two-day 
conference  agenda  was  organized  Into  panels  of  presentations  chaired  by  a 
civilian  expert,  system  demonstrations,  small  group  discussion  sessions  and 
presentations,  and  a  conference  summary.  (The  conference  agenda  can  also 
be  found  in  the  Appendix.) 

The  conference  was  opened  by  Dr.  Harold  F.  O'Neil,  Or.,  Director  of 
the  Army  Research  Institute's  Laboratory  for  the  Behavioral  and  Social 
Sciences.  He  Indicated  that  the  problem  stimulating  the  Army's  Interest  In 
automated  author  aiding  systems  is  the  lack  of  efficient  methods  for 
creating  large  amounts  of  computer-assisted  Instructional  (CA1)  materials 
and  the  failure  of  Instructional  System  Development  (ISD)  procedures  to 
yield  high  quality,  effective  Instructional  materials.  He  framed  the 
objective  of  the  conference  as  a  state  of  the  art  assessment  of  online  aids 
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to  create  online  Instruction,  focusing  on  the  questions  of:  What  do  we 
currently  know?  and  What  should  we  do?  Dr.  O'Neil  identified  the  eventual 
products  of  the  conference  as  an  Automated  Authoring  System  for  CAI,  a 
research  agenda  for  Incorporating  artificial  Intelligence  techniques  and 
methods  into  automated  authoring  aids,  and  the  revision  of  ISD  policies  and 
procedures.  (A  copy  of  Dr.  O'Neil's  briefing  charts  are  included  In  the 
Appendix , ) 

The  following  sections  of  this  report  summarize  t  .  comments  of 
conference  presenters.  They  are  organized  around  the  five  panels  centered 
on  particular  aspects  or  features  of  automated  author  aiding  systems. 
Session  1,  chaired  by  Dr.  Walter  Dick  of  Florida  State  University  served  as 
an  overview  of  the  range  of  Issues  and  examined  the  potential  of 
computer-based  Instruction  for  Improving  Instructional  quality-  Session 
II,  chaired  by  Dr.  M.  David  Merrill  of  the  University  of  Southern 
California,  addressed  general  Issues  In  the  design  of  automated  authoring 
aids.  Session  III,  chaired  by  Dr.  Robert  Davis  of  the  University  of 
Illinois,  concerned  specific  automated  aids  for  Instructional  design. 
Session  IV,  chaired  by  Dr.  Thomas  Duffy  of  Carnegie  Mellon  University, 
focused  on  Issues  in  authoring  aids  for  document  design.  Session  V, 
chaired  by  Dr.  Wallace  Fuerzeig  of  Bolt,  Beranek,  and  Newman,  examined 
future  directions  for  automated  author  aiding  systems  Including 
applications  of  artificial  Intelligence.  As  a  conclusion,  a  summary  of  the 
small  group  discussions  and  the  overall  conference  Is  provided.  (Hard 
copies  of  the  vlewgrapbs  used  by  presenters  may  be  requested  from  CSE  or 
from  the  authors  directly.) 
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Session  I:  Computer-Based  Instruction:  VI1 1  It  Improve  Instructional 
Quality?  by  William  Montague  and  Wallace  Wulfeck,  Navy  Personnel  Research 
and  Development  Center. 

The  use  of  modern  computer-based  gadgets  for  Instructional  purposes  Is 
being  advocated  widely.  This  advocacy  Is  based  mostly  on  the  assumption 
that  computer-based  Instruction  can  solve  many  of  the  nation's  educational 
problems.  In  newspapers,  magazines,  and  on  television,  advertisements 
suggest  that  computer-based  Instruction  Is  effective,  fun,  can  answer 
concern  for  the  "decline"  In  educational  quality,  and  provide  business, 
Industry,  and  the  military  efficient  and  effective  means  of  training 
personnel.  Although  use  of  this  gadgetry  will  continue  to  Increase  simply 
because  of  Its  availability  and  declining  costs,  we  are  dubious  about  the 
assumption  or  hope  that  Its  presence  will  Improve  the  quality  of  education 
and  training,  at  least  over  the  near  term.  It  Is  not  that  computer-based 
Instructional  (C6I)  systems  cannot  be  effective;  they  can  be,  and  some 
are.  Recent  reviews  thoroughly  summarize  many  demonstrations  of  the 
effectiveness  gains  but,  while  affordable  hardware  Is  a  necessary 
Ingredient  for  widespread  effectiveness  of  CBI,  It  Is  not  sufficient. 
Several  other  Ingredients  are  necessary:  good  Instructional  design  which 
uses  computer  power  In  appropriate  ways,  supportable  and  transportable 
software,  and  attention  to  the  ongoing  Instructional  systems  Into  which  CBI 
may  be  Inserted. 

We  believe  that  Improvement  of  Instruction  through  CBI  will  be  a 
relatively  slow,  evolutionary  process.  Reasons  for  this  are  that;  (1) 
Instructional  quality  Is  difficult  to  achieve  regardless  of  the  method  of 
delivery,  (2)  computers,  as  Instructional  tools,  are  In  a  rudimentary  state 
of  development,  (3)  Improvements  In  either  instructional  design  or 
computer-based  delivery  will  depend  on  fundamental  changes  In  the 
scientific  base,  and  (4)  systematic  planning  for  acquiring,  standardizing 
and  distributing  proven  Instructional  programs,  and  for  Incorporating  them 
Into  schooling,  has  not  been  done. 

First,  we  review  some  attempts  to  improve  quality  through  the 
development  and  Implementation  of  systematic  approaches  to  Instructional 
design.  These  attempts  have  not  been  very  successful,  and  their  refinement 
will  take  time.  Second,  we  briefly  review  some  developments  that  have  led 
to  common  forms  of  CBI  and  show  that  the  advantage  of  using  CBI  Is  often 
unclear.  Third,  we  show  that  the  problems  with  both  Instructional  design 
and  with  traditional  CBI  are  due  largely  to  shortcomings  In  the  underlying 
scientific  base  of  the  psychology  of  learning  and  instruction.  Fourth,  we 
describe  developments  in  cognitive  and  Instructional  psychology  and  In 
computer  science  and  artificial  Intelligence  which  provide  hope  that  a 
better  scientific  base  will  develop;  and  we  describe  particular  CBI  systems 
which  have  been  built  on  these  developments.  Finally,  we  discuss  some 
things  that  can  be  done  to  provide  a  means  for  the  widespread  distribution 
and  life-cycle  support  of  CBI  systems  that  have  been  found  to  be  effective 
for  specific  purposes. 

What  Needs  to  be  Done  to  Obtain  Quality  Improvement?  We  repeatedly 
make  the  point  that  computers  and  CBI  software  should  not  be  bought  with 
the  expectation  that  they  will  solve  today's  instructional  problems. 
Instead,  CBI  programs  must  be  built  explicitly  to  teach,  and  this  means 
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that  some  form  of  ISD  must  be  performed.  That  Is  why  this  presentation 
Includes  both  ISO  and  CBI  as  topics;  for  quality,  they  can't  be  separated. 
Moreover,  If  the  quality  of  ISD  Is  not  improved,  then  CBI  will  only  Improve 
through  sporadic  artistry,  Therefore,  in  our  view,  two  developments  are 
necessary:  automated  aids  for  ISD  and  a  coherent  software  distribution  and 
maintenance  system. 

Automated  Aids  for  Instructional  Design  and  Development.  The 
continued  lack  of  progress  with  ISD  Implementation  seems  due  to  the 
variable  quality  of  those  doing  the  Implementation  and  the  lack  of  "how  to 
do-  It"  procedures  In  usable  forms.  Since  added  procedural izatl on  Is 
unlikely  to  be  useful,  It  Is  necessary  to  provide  job  aids.  In  paper  form 
job  aids  can  help,  but  too  much  still  depends  on  learning  by  the 
developers,  and  there  Is  little  time  or  resources  for  this.  Computer-based 
aids  to  authoring  Instruction  can  make  substantial  differences  In  the 
quality  of  Instruction  whether  it  is  on-line  or  off-line  and,  of  course,  In 
the  efficiency  of  ISD. 

The  ISD  process  has  formidable  record-keeping  problems.  A  typical 
military  training  program  has  hundreds  or  sometimes  even  thousands  of 
learning  objectives  which  must  be  developed,  cross-referenced,  tested  and 
taught.  Records  also  cover  a  wide  variety  of  other  ISD  activities  such  as 
generating  test  Items,  choosing  alternative  training  media  and  strategies, 
evaluating  graduates,  and  revising  courses. 

More  Importantly,  computerized  ISD  aiding  systans  can  not  only  assist 
with  these  record-keeping  problems,  they  can  facilitate  the  development 
process  Itself  by  providing  guidance  for  test  and  Instructional  development 
and  similar  tasks.  Moreover,  computer-based  systems  can  ensure  that 
guidance  Is  followed  by  monitoring  and  evaluating  developers'  performance, 
especially  by  forcing  attention  to  the  delivery  options  available  and  the 
trade-offs  among  them,  and  by  assisting  developers  as  they  proceed. 

Computer  systems  can  also  provide  training  for  Instructional  developers  who 
can  fit  It  Into  their  work  schedules.  Finally,  these  systems  are  essential 
for  aiding  Implementation  and  utilization  of  CBI.  Today,  most  CBI  users 
such  as  teachers,  school  boards,  military  training  activities  and 
businesses  want  the  ability  to  customize  and  adapt  Instructional  software. 
They  cannot  do  this  unless  tools  are  available  for  modification  and 
refinement  of  CBI  programs. 

The  most  desirable  reason  for  using  computer  tools,  however,  Is  that 
the  modern  design  and  development  tools  can  be  Incorporated  Into  the  ISD 
process.  This  provides  the  only  mechanism  hereby  developing  science  can 
Improve  day-to-day  practice.  Computer-based  author  aids,  then,  must  be 
developed  to  support  both  ISD  for  CBI  and  utilization  of  CBI. 

Improved  Distribution  and  Maintenance  of  CBI  Programs.  The  transfer 
of  software  Is  a  problem.  There  are  attempts  to  catalog  programs  and 
annotate  them.  While  these  efforts  serve  to  acquaint  people  with  the  range 
of  programs  available,  several  major  problems  remain.  First,  many  schools, 
businesses  and  military  agencies  are  buying  computers.  Unfortunately, 
these  are  often  not  compatible,  so  common  software  packages  cannot  be 
developed  and  shared  without  substantial  recoding  and  duplication  of 
effort.  A  second  problem  Is  the  tremendous  repetitive  nature  of  the 
programming  efforts.  One  programmer  or  another  develops  programs  that 
present  questions  and  require  selected  answers  or  fill  In  answers,  present 
text  descriptions,  etc.  All  of  these  require  functionally  the  same  code. 
Computer  software  companies  have  recognized  the  same  sort  of  repetitiveness 
In  computer  programs  and  are  developing  ways  to  use  codes  already  developed 
In  new  programs.  This  speeds  up  the  development  process  and  reduces  errors 
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by  a  large  factor.  A  third  problem  is  that  most  current  CBI  programs  are 
not  supported  by  appropriate  authoring  support  and  instructional  management 
aids. 

We  believe  that  a  solution  to  these  problems  Is  to  develop  families  of 
CBI  software  to  support  computer-based  Instruction  in  a  wide  variety  of 
education  applications.  This  can  be  done  by  developing  libraries  of 
computer-based  Instructional  programs,  sufficiently  flexible  to  support 
development,  delivery  and  management  to  meet  any  instructional 
requirements.  The  library  should  also  be  concerned  with  demonstration  of 
and  specifications  for  generic  hardware  systems  capable  of  executing 
library  software,  and  with  planning  for  and  assisting  Institutionalization 
of  CBI  programs.  By  providing  transportable,  carefully  tested  CBI  software 
and  development  tools,  compatibility  and  supportablll  ty  problems  are 
solved,  user  requirements  are  more  efficiently  addressed,  implementation 
and  life-cycle  costs  are  reduced,  standard  data  on  student  performance  and 
CBI  cost  effectiveness  can  be  obtained  for  budget  justification,  and 
acquisition  costs  of  training  can  be  reduced.  And  again  most  important, 
Institutional  software  libraries  can  achieve  a  “critical  mass*  so  that 
evolutionary  improvements  through  application  of  new  technologies  like 
authoring  aids  can  be  achieved. 


Session  II:  General  Issues  In  the  Design  of  Automated  Authoring  Aids 

The  three  papers  In  this  session  examined  critical  Issues  In  the 
design  of  automated  authoring  aids  for  developing  Instruction  and 
assessment.  Dr.  Marken  highlighted  the  time  and  resource  constraints  under 
which  training  development,  particularly  by  private  contractors,  typically 
occurs  and  their  Implications  for  authoring  aids.  Dr.  Watson  stressed  the 
Importance  of  considering  subject  matter  content  and  learner 
characteristics  In  designing  effective  computer-based  training.  Dr.  Rold 
discussed  the  use  of  computerized  Item  writing  aids,  adaptive  and 
diagnostic  testing  methods,  and  automated  test  Interpretation  systems  to 
assist  In  the  design  and  use  of  assessment. 


Automated  Instructional  Development  Aid  (AIDA)  by  Mara  Marken,  Nor thru p 
Corporation 

Thousands  of  training  courses  are  developed  annually  In  the  military 
and  Industrial  communities.  Most  often  these  programs  are  developed  by 
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subject  matter  experts  (pilots,  mechanics,  accountants,  etc.)  who  have  no 
background  in  instructional  design  methodology.  This  Inexperience,  coupled 
with  constraints  of  short  development  lead  times  and  limited  funds  and 
manpower,  make  training  development  a  difficult  and  Inefficient  enterprise. 

The  automated  Instructional  development  aid  (AIDA)  was  designed  to 
meet  the  needs  and  skill  level  of  subject  matter  experts.  It  maximizes  the 
use  of  their  technical  knowledge  while  minimizing  the  demand  for  expertise 
In  Instruction  design.  Simple  procedures  are  easy  to  learn  and  the 
tutorial  software  enables  the  user  to  conduct  a  task  analysis,  determine 
media  requirements,  and  develop  lesson  plans  without  a  formal  training 
course. 

AIDA  acknowledges  both  the  constraints  of  the  political /economic 
environment  and  the  tendency  for  subject  matter  experts  to  base  decisions 
on  Intuitive  feelings  and  past  experience.  These  liabilities  are 
transformed  Into  assets  by  encouraging  users  to  express  their  frustrations 
and  Intuition.  This  fosters  creativity  while  providing  documentation  for 
further  analysis  and  Informed  decision  making. 

AIDA  can  be  used  In  conjunction  with  many  of  the  other  automated 
systems  now  available  although  1t‘s  focus  on  lesson  plan  development  Is 
unique.  It  automatically  produces  performance  objectives,  course  outlines, 
training  standards,  course  charts,  plans  of  Instruction,  Instructor 
supplements  and  media  specification. 


Effective  Learning  and  Computer-Based  Training  (CBT)  by  Marcia  Watson, 

Xerox  Learning  Systems 

As  with  any  Instruction,  computer-based  training  (CBT)  needs  to  take 
Into  consideration  both  learner  and  subject-matter  Issues.  However,  In  the 
case  of  CBT  as  an  Individually  delivered  form  of  Instruction,  the  Inclusion 
of  both  learner  and  content  Issues  Is  especially  important.  Content  needs 
to  be  carefully  analyzed  for  processes  required  for  learners  to  acquire  and 
utilize  subject-specific  data,  subject-specific  algorithms,  and 
transferable  concepts  and  algorithms.  Learner  issues  that  should  be 
considered  in  designing  CBT  Include  emotional  set  and  types  of  feedback 
(social,  structural).  Both  learner  and  content  Issues  need  to  be 
considered  If  effective  CBT  Is  to  be  designed,  In  that  learners  with 
positive  emotional  sets  and  appropriate  feedback  will  learn  more 
(especially  In  Independent  learning  situations)  than  learners  who  must 
attend  to  their  emotional  hemostasis  and  for  whom  feedback  Is  Irrelevant. 

Subtly  Important  to  the  design  of  CBT  are  top-down/bottom-up  Issues  In 
Instructional  design  and  learning.  Instructional  designers  will  often  take 
the  approach  that  It's  simple  to  learn  a  given  topic  because  the 
instructional  designer  has  access  to  the  subject  matter  as  a  whole  and  can 
see  it  in  an  organized  pattern.  It  takes  effort  on  the  instructional 
designer's  part  to  keep  the  learner's  perspective  of  relative  overload  and 
confusion  In  mind.  The  designer  also  needs  to  remember  that  learners  will 
seek  structure  In  the  discrete  experiences  Involved  In  their  learning. 
Further,  it  is  wise  to  help  shape  learners'  views  of  the  topic's  structure 
so  as  to  maximize  the  probability  of  functional  concepts  being  formed  that 
will  hold  water  In  a  variety  of  situations.  Finally,  It  is  important  for  a 
designer  to  maximize  learners'  focus  on  content  and  minimize  their 
attention  to  hardware/software  mechanics. 

In  short:  (1)  time  ''wasted1*  on  learner  feelings  yields  performance 
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results;  (2)  knowing  learners'  natural  reward  structures  helps  In  designing 
feedback;  (3)  it  Is  more  productive  to  aid  and  abet  learners'  search  for 
meaning  in  content  thatn  to  Ignore  It;  and  (4)  CBT  Is  most  effective  when 
its  design  considers  both  learner  and  content  factors. 


Item  Writing  and  Test  Interpretation  by  Gale  H.  Roid,  Western 
Psychological  Services. 

Three  areas  of  research  and  development  that  should  be  monitored  by 
designers  of  author-aiding  systems  are;  (1)  computerized  Item-writing  aids, 
(2)  adaptive  and  diagnostic  testing  methods,  and  (3)  automated 
test-interpretive  systems. 

Item  Writing.  Two  trends  can  be  seen  In  Item-writing  methods:  (a)  a 
movement  toward  diagnostic  testing  and  test  scoring  based  on  the  strategies 
used  by  examinees  In  solving  items  of  the  problem-solving  type,  and  (b) 
Increased  Interest  In  the  dimensionality  of  tests  with  the  advent  of  factor 
analysis  for  dichotomous  Items.  Author  aids  could  Incorporate  routines  for 
diagnostic  scoring  (examining  patterns  of  wrong  answers),  and  aids  for 
assessing  dimensions  underlying  Item  pools.  Also,  Item  readability 
analyses,  the  "ambiguity"  Index  proposed  by  Goldberg,  and  the  new 
domain-referenced  test  software  recently  developed  by  Nitko  and  associates 
at  Pittsburgh  should  be  Investigated,  and  an  automated  "test  critic"  with 
Item  analysis  developed. 

Adaptive  Testing.  Aids  for  authors  developing  adaptive  testing  based 
on  ISb  models  have  been  slow  In  emerging,  perhaps  due  to  the  technical 
nature  of  the  mathematical  models  underlying  these  tests.  However,  with 
Rasch  software  and  software  that  can  Implement  estimators  of  3-parameter 
models  appearing  now  for  microcomputers,  adaptive  testing  may  finally  be 
feasible. 

Computerized  Test  Interpretation.  In  the  field  of  psychological 
assessment,  a  wide  variety  of  techniques  and  software  are  developing  for 
the  automated  Interpretation  of  patterns  of  test  scores,  resulting  In 
elaborate  narrative  reports  for  Individual  examinees.  To  my  knowledge, 
these  methods  have  not  been  applied  to  achievement  testing  or  computerized 
test-like  exercises  that  are  embedded  in  computer  assisted  Instruction. 
However,  the  potential  Is  great  for  producing  printed  reports  for 
Instructors  that  "talk  to  them"  about  the  performance  of  Individual 
trainees.  There  are  at  least  three  levels  of  Interpretive  reports:  (a) 
descriptive,  which  has  logical  rules  such  as  "this  score  Is  above  average", 
(b)  "diagnostician  modeled"  in  which  the  Interpretive  rules  of  a  skilled 
Instructor/evaluator  are  automated  so  that  the  resulting  printed  report 
reads  In  a  way  that  the  Instructor  would  have  written  It  If  done 
Individually,  and  (c)  fully  "actuarial"  in  which  research-based  decision 
rules  (quantitative,  not  qualitative)  are  automated.  The  "actuarial" 
method,  similar  to  that  used  for  the  MMPI  and  other  heavily  researched 
instruments,  would  make  predictive  statements  such  as  "this  student  may 

have  difficulty  with  the  next  course,  given  that  he  failed  the  _ 

-type  items  In  this  program-"  (where  research  has  shown  a  correlation 
EeTween  performance  on  the  CAI  lessons  and  future  courses). 
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Session  III;  Automated  Aids  for  Instructional  Design 

The  five  presentations  In  this  session  concerned  specific  systems  that 
have  been  designed  to  aid  authors  In  Instructional  development.  Dr. 
Goldberg  described  the  use  of  a  software  engineering  tool,  Structured 
Design  and  Documentation  Language  (SDDL) •  to  produce  general  lesson  design 
schemas  which  can  be  applied  across  lessons  having  similar  learning 
objectives.  Dr.  Dural  1  discussed  the  features  of  the  PLATO  Author  System 
and  Its  capabilities  as  an  automated  authoring  aid  for  Instructional 
designers.  Mr.  Bauer  described  a  project  to  develop  a  computer  capability 
using  engineering  specifications  to  develop  task  analyses  for  the  operation 
and  maintenance  of  equipment  by  Individual  soldiers  which  can  serve  as 
front  end  analyses  for  Instructional  development.  Dr.  Akscyn  described  the 
ZOG  system  which  Is  a  general-purpose,  human -computer  Interface  system 
which  has  been  used  as  a  training  system  as  well  as  for  database  systems, 
management  Information  systems,  document  management,  software  management, 
and  electronic  communication.  Mr.  Morriss  described  the  Maintenance 
Information  Authoring  System  (MIAS),  a  software  development  system  for 
creating  an  accurate  and  complete  data  base  of  procedural  Information  which 
can  be  used  In  instructional  delivery  and  job  aiding  systems. 


Application  of  Software  Development  Tools  to  Courseware  Authoring  by 
Stephen  L.  Goldberg,  Army  Research  Institute 

Custom  computer-based  training  courseware  that  deviates  significantly 
from  text-based  didactic  Instruction  will  be  expensive  and  time  consuming 
to  produce  given  current  authoring  aids.  The  Army  Research  Institute  and 
Scientific  Systems,  Inc.  have  the  problem  of  producing  a  large  amount  of 
courseware,  In  a  short  period  of  time  with  limited  resources.  We  decided 
to  look  toward  methods  being  used  in  software  engineering  for  the  means  to 
make  the  courseware  authoring  process  more  efficient  arid  less  costly. 

In  most  software  development  environments,  when  a  series  of  similar 


programs  must  be  written,  a  general  program,  called  a  program  schema,  Is 
developed.  Different  versions  of  the  general  program  are  then  produced, 
and  great  efficiencies  can  result.  In  effect  the  computer  is  doing  most  of 
the  detail  work. 

Good  software  designers  differ  significantly  In  their  approach  from 
novices.  Experts  are  much  more  likely  to  take  an  approach  that  Is  top-down 
and  hierarchical,  rather  than  solving  little  pieces  of  the  problem;  breadth 
first,  so  that  all  problems  at  a  given  level  In  a  hierarchy  are  solved 
before  going  to  the  next  level;  modular,  such  that  experts  wind  up  with 
quasi-independent  "packages"  Instead  of  highly  Interdependent  subdivision; 
and  reducible  to  solutions  that  use  existing  code  or  algorithms. 

In  looking  at  the  Instructional  design  process,  we  decided  that  It  too 
lent  Itself  to  a  top-down,  breadth-first  modular  approach  that  results  in 
general  lesson  design  schemas  which  could  be  applied  across  lessons  having 
similar  learning  objectives.  A  software  engineering  tool.  Structured 
Design  and  Documentation  Language  (SDDL)  was  used  to  facilitate  the  design 
process  and  produce  documents  that  communicated  lesson  flow  and  parameter 
requirements  to  courseware  developers.  Each  module  In  SDDL  represents  a 
section  of  code  that  may  be  developed  without  worry  about  how  It  will  fit 
Into  the  entire  lesson.  Once  a  module  Is  coded  and  running,  other  modules 
which  Interact  with  them  are  easier  to  code  and  debug. 

Within  the  courseware  development  project  currently  underway  a  number 
of  lesson  schemas  have  been  written.  Each  Incorporates  a  lesson  design 
strategy  that,  with  small  variations,  can  be  applied  to  a  number  of  lesson 

segments  that  serve  the  same  function  for  different  systems  or  pieces  of 

equipment.  The  use  of  the  lesson  schemas  allows  for  quick  generation  of 
specific  lessons  through  Insertion  of  appropriate  content. 

The  courseware  that  results  from  the  process  described  above  has  the 

added  advantage  that  students'  Interactions  with  each  type  of  lesson  Is 
standardized  so  that  the  student  can  concentrate  on  the  lesson  content  and 
not  his  Interaction  with  the  computer.  It  Is  currently  an  open  question 
whether  Intensive  Interaction  with  a  training  program  produced  In  this  way 
will  result  In  student  boredom.  For  the  present  program  In  which  reserve 
soldiers  will  Interact  with  the  system  for  at  most  4  hours  a  month  the 
question  is  moot. 


The  PLATO  Authoring  System  by  Edwin  P.  Durall,  Control  Data  Corporation 

The  PLATO  Author  System  and  Its  capabilities  an  an  Automated  Authoring 
Aid  are  described.  The  PLATO  Author  system  uses  the  concept  of  the  Lesson 
Model  In  reducing  the  cost  and  complexity  of  computer-based  education  (CBE) 
courseware  development.  A  Lesson  Model  Is  a  software  utility  that  provides 
certain  features  Including: 

-  predetermined  Instructional  strategies 

-  advanced  graphics  and  text  editors 

-  direct-display  editing 

-  preformatted  screen  displays  in  basic  models 

-  menu/prompt  selection  of  branching  and  judging  options 

-  "programmer! ess"  data  entry  features 

-  color,  animation,  and  videodisc  capabilities  in  more  advanced 
models 

The  PLATO  Author  System  currently  consists  of  two  sets  of  authoring 
utilities:  PLATO  Courseware  Development  (PCD1)  and  PLATO  Courseware 
Development  and  Delivery  (PCD2). 
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PC01  Is  a  family  of  totally  progranmerless  Lesson  Models  that  offers 
microcomputer-based  authoring  and  delivery,  l.e.,  neither  authoring  nor 
delivery  requires  access  to  central  PLATO.  Seven  Lesson  Models  are 
available  with  a  wide  variety  of  flexibility  and  ease-of-use  features. 

PCD2  requires  access  to  central  PLATO  for  authoring,  but  allows 
lessons  to  be  downloaded  to  disks  for  delivery  on  a  variety  of 
microcomputer  systems.  PC02  offers  more  flexibility  than  PCD1  In  that  In 
addition  to  certain  progranmerless  features,  the  full  range  of  capabilities 
of  the  PLATO  Author  Language  and  the  micro  PLATO  Author  Language  (formerly 
known  as  TUTOR)  are  available  through  PCD2. 

Further  enhancements  to  the  PLATO  Author  System  to  offer  automated 
authoring  aids  In  the  area  of  CBE  courseware  design,  as  well  as  development 
and  delivery  ,  are  anticipated  In  the  near  future. 


Experience  with  the  ZOG  Human-Computer  Interface  System  by  Robert 
Akscyn,  Carnegl e-Mellon  University. 

ZOG  Is  a  general-purpose,  human  computer  interface  system  based 
primarily  on  the  concept  of  menu-selection,  with  a  large  database  of  menus 
and  rapid  response  to  selections.  ZOG  Is  Intended  to  be  used  by  both 
novice  and  expert  users,  providing  a  single  Interface  mechanism  that 
Integrates  all  the  computer  functions  needed  by  the  user. 

The  basic  unit  of  representation  in  ZOG  Is  called  a  frame. 

Originally,  the  notion  of  a  frame  meant  a  “structured  screenful",  l.e., 
everything  the  user  could  see  on  the  terminal  screen  at  one  time. 

Nowadays,  with  the  advent  of  high-resolution  screens,  Implementations  of 
ZOG  provide  for  several  frames  to  be  displayed  simultaneously  on  the 
screen.  A  ZOG  frame  consists  of  a  set  of  items  of  different  types,  each  of 
which  carries  Its  own  positioning  Information. 

There  are  three  types  of  Interactions  with  ZOG:  navigation.  Invoking 
programs,  and  editing.  The  default  mode  of  Interaction  Is  navigation,  In 
which  the  user  makes  a  selection  via  the  keyboard  or  pointing  device 
(mouse),  and  the  system  then  responds  by  displaying  the  next  frame.  Most 
selections  lead  to  other  frames,  but  some  have  "actions",  which  perform  a 
procedural  function  such  as  running  a  particular  program.  Finally,  the 
user  can  enter  the  frame  editor  at  any  frame  and  make  changes  to  the  frame 
(If  he  has  the  requisite  privileges,  protection  being  Implemented  at  the 
level  of  the  Individual  frame). 

The  ZOG  system  Is  based  on  a  broad  set  of  principles  about  desirable 
features  of  a  human-computer  Interface  Including: 

-  total  environment 

-  flexible,  efficient  tool 

-  direct  manupulation  of  data 

-  semi-automatic  operation 

-  low  learning  overhead 

-  safe,  exploratory  environment. 

A  major  component  of  a  ZOG  system  is  Its  database,  which  Is  somewhat 
different  from  the  traditional  notion  of  a  database.  The  following 
principles  govern  the  design  of  the  database: 

-  large  size 

-  shared  by  multiple  users 

-  generality  of  representation 

-  network  structures 
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-  tree  structures 

-  menus. 

ZOG  also  makes  a  major  commitment  to  a  particular  style  of  Interaction 
between  the  user  and  the  system,  as  indicated  by  the  following  features: 

-  menu  selection 

-  fast  response 

-  browsing 

-  active  selections 

-  no  hidden  selections 

-  common  contnan  s 

-  editor 

-  no  scror-'  . 

A  ZOG  system  needs  more  than  just  a  database  with  an  Interface  --  It 
needs  some  mechanism  for  extending  the  system  to  provide  new  functions  for 
the  user.  The  following  features  describe  how  this  Is  done: 

-  mapping  data  structures 

-  embedded  programs 

-  environment  frames 

-  frames  for  Input/output. 

ZOG  has  been  used  In  a  variety  of  applications,  Including 
instruction/training.  ZOG  has  been  used  as  a  training  system  in  several 
capacities:  (1)  providing  on-line  help;  (2)  as  a  guidance  system  for  using 
other  on-line  systems  by  shielding  users  from  the  Idiosyncracles  of  a 
particular  Interface;  and  (3)  as  an  Index  and  control  mechanism  for  a 
videodisc  player. 

We  believe  that  ZOG  has  many  strong  points;  In  fact,  It  has  often 
surprised  us  (even  with  our  understandable  optimism)  how  well  ZOG  has 
adapted  to  the  many  demands  we  have  made  of  It.  The  most  Important  of  the 
strong  points  are  described  below: 

-  robust  enough  to  be  put  Into  operational  use 

-  easy  for  computer  novices  to  learn  and  use 

-  users  do  not  outgrow  ZOG  as  they  become  expert 

-  can  assimilate  and  Integrate  many  different  applications 

-  supports  database  browsing 

-  supports  large  databases 

-  can  exploit  schemas  for  building  databases 

-  can  be  used  as  the  sole  Interface  (shell)  for  an  operating  system 

-  has  a  simplified  model  of  window  use 

-  has  a  simplified  model  of  multiple  processes 

-  can  be  used  as  an  Interface  mechanism  for  video  discs 

-  can  make  good  use  of  a  pointing  devise  (mouse) 

-  can  exploit  distributed  systems 

-  supports  a  community  of  communicating  users. 

Below  are  the  weaknesses  of  ZOG  that  we  have  become  aware  of: 

-  ZOG  sacrifices  efficiency  of  particular  applications  to  get 
Integration 

-  ZOG  does  not  support  a  fast  database  query  language 

-  inexperienced  users  can  get  lost 

-  biased  too  much  toward  breadth-first  view 

-  ZOG  depends  too  critically  on  the  speed  of  the  disk  technology 

-  can't  represent  all  states  of  a  complex,  unordered  task  envlronnent 

-  can't  handle  highly  dynamic  data 

-  lack  of  graphics  and  multiple  fonts 

-  ZOG  cannot  be  used  over  standard  telecommunication  lines. 
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There  Is  now  work  going  on  outside  CMU  to  create  a  follow-on  to  ZOG  called 
KMS  (Knowledge  Management  System).  KMS  Is  designed  to  remedy  many  of  the 
shortcomings  of  the  ZOG  Implementations,  and  specifically  to  make  full  use 
of  the  potential  of  high-resolution  display  technology.  Some  of  the 
features  of  KMS  that  represent  Improvements  over  ZOG  are  the  following: 

-Graphics  (lines,  rectangles,  curves,  picture  Images)  and  multiple 
fonts. 

-Greater  use  of  the  pointing  device  to  specify  objects  and  parts  of 
objects  to  operate  on,  and  screen  cursor  Images  to  provide  feedback  of  the 
current  system  context. 

-Direct  output  of  good  quality  hardcopy  (no  need  for  a  separate 
formatting  system  that  operates  as  a  post-processor). 

-Copying  material  easily  across  frame  boundaries. 

-Additional  Intrinsic  views  of  the  database  --  specifically,  a 
depth-first  view. 

-Closer  Integration  of  the  editor  with  the  rest  of  system,  to  make 
editing  seem  more  natural,  rather  than  a  special  mode  that  one  1$ 
continually  entering  and  leaving. 


The  Maintenance  Information  Authoring  System  by  Stephen  Morrlss,  Texas 
Instruments. 

The  Maintenance  Information  Authoring  System  (MIAS)  Is  a  software 
development  system  dedicated  to  the  construction  of  accurate  and  complete 
procedural  Information,  for  use  by  noncomputer  knowledgeable  technical 
writers,  that  conforms  to  the  special  requirements  and  constraints  Imposed 
by  an  equally  dedicated  delivery  medium.  This  paper  describes,  beginning 
with  the  requirements  and  constraints  of  the  delivery  medium,  the  process 
and  structure  of  authoring  system  specific  maintenance  Instructions. 

In  1979,  the  Department  of  Defense,  sponsored  the  development  of  a  new 
way  to  deliver  Information  to  military  technicians.  The  Intent  of  this  new 
system  was  to  provide  Individuals  with  a  portable  tool  with  vAilch  the 
Information/knowledge  requirements  to  operate  and  maintain  the  new  weapon 
systems  could  be  obtained  at  the  time  and  place  the  Individual  needed  It. 
The  base  level  requirements  for  this  new  information  delivery  medium  were 
that  It  be  man  portable,  rugged,  easy  to  use,  and  easy  to  update  the 
Information  delivered. 

Our  response  to  this  challenge  was  the  Personal  Electronic  Aid  for 
Maintenance  (REAM).  REAM  Is  a  suitcase  size  computer,  weighing  near  25 
pounds.  The  suitcase  contains  a  detachable  man/machine  Interface  that 
contains  a  5  by  7  Inch  flat  panel  electroluminescent  display.  The  display 
Is  capable  of  matrix  graphics  on  a  352  by  512  grid.  The  line  density  Is  /l 
lines  per  Inch.  The  system  has  the  capacity  to  read  text  aloud  by  means  of 
text-to-speech  synthesis  and  to  respond  to  voice  command  by  the  system 
operator.  Information  storage  Is  by  means  of  1  megabyte  solid  state  CMOS 
static  RAM  cartridges.  The  system  Is  capable  of  battery  operation  for  up 
to  4  hours  and  Is  compatible  to  standard  110  current.  The  system  Is 
ruggedlzed  to  withstand  hostile  environments  Including  moisture  and 
temperature  extremes. 

The  Information  storage,  being  limited  by  the  size,  envi rormental ,  and 
power  constraints,  required  the  development  of  a  highly  specialized 
database  design  that  would  allow  for  t  e  compression  of  the  equivalent  of 
3-4000  pages  of  text  and  graphics  within  the  megabyte  cartridge. 
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At  the  same  time  we  were  developing  the  PEAM  system,  we  began  the 
design  of  a  method  to  develop  the  Information  to  be  delivered.  Because 
PEAM  was  considered  an  Instructional  delivery  system,  our  starting  point 
was  the  existing  CAI  authoring  systems  currently  In  use. 

As  we  progressed  In  our  design.  It  quickly  became  apparent  that  there 
existed  a  marked  difference  In  the  methods  used  In  CAI  authoring  and  the 
type  of  Information  that  we  Intended  for  PEAM.  Specifically,  CAI  systems 
are,  by  design,  generic  tools  for  mapping  information  Into  one  or  many 
learning  strategies.  The  tools  must  be  general  purpose,  because  the 
domains  of  knowledge  or  the  subject  matter  vary  In  their  structure,  their 
completeness,  their  vocabularies  and  their  level  of  abstraction. 

PEAM  Information,  on  the  other  hand,  Is  composed  of  only  one  kind  of 
learning  strategy,  step  by  step  Instruction.  The  Information  structure  and 
the  degree  of  completeness  Is  fixed  by  the  structure  of  the  system  to  be 
maintained.  The  vocabulary,  we  discovered,  Is  relatively  constrained, 
containing  approximately  1300  generic  terms  (exclusive  of  part  names).  The 
level  of  detail  had  to  be  at  a  performance  or  action  level.  Graphic  data 
has  two  functions:  1)  to  provide  for  location  discrimination;  and  2)  to 
provide  for  Identification  of  specific  parts. 

We  designed  an  authoring  systeem  that  is  capable  of  performing  to  the 
following  requirements: 

1.  Takes  full  advantage  of  the  Inherent  structure  of  the  maintenance 
domain  by: 

A.  Providing  assistance  In  defining  the  system  and  Information 
structure. 

B.  Providing  assistance  In  the  control  of  system  and  generic 
vocabulary,  syntax,  and  thus,  the  level  of  detail  for  each 
Instruction. 

C.  Providing  assistance  In  development  of  the  troubleshooting 
Information. 

2.  Takes  full  advantage  of  available  source  documentation  by: 

A.  Maintaining  source  document  and  graphic  data  control. 

B.  Allowing  for  direct  Information  capture. 

C.  Allowing  the  easy  trace  of  Information  change  based  upon  changes 
In  the  engineering  of  the  system. 

3.  Takes  full  advantage  of  the  structure  for  authoring  by: 

A.  Providing  for  assistance  In  setting  up  the  job. 

B.  Providing  assistance  In  management  of  the  job. 

C.  Provides  for  the  most  skill  at  the  highest  and  less  burdened 
activities  of  the  job  and  the  least  skills  at  the  lowest  and  more 
numerous  tasks  of  the  job. 

The  objectives  for  MIAS  Include: 

-  To  provide  a  usable  system  for  primarily  defense  contractors. 

-  To  provide  for  low  cost  acquisition  and  operation. 

-  To  provide  for  a  flexible  user  base  from  experienced  to 
Inexperienced  writers. 

-  To  provide  for  consistent  quality  products. 

-  To  be  easy  to  learn  and  use. 

-  To  provide  management  of  the  ability  to  observe  the  total  picture  of 
the  job  process. 

-  To  provide  for  product  cost  control. 
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Session  IV:  Issues  In  Authoring  Aids  for  Document  Design 

The  papers  In  this  session  examined  Issues  In  aiding  systems  for 
authors  of  documents.  These  Issues  and  systems  are  relevant  to  the  problem 
of  Instructional  design  since  a  great  deal  of  Instruction  both  offline  and 
online  Is  text-based.  Mr.  Braby  described  computer  aided  authoring 
routines  to  create  Procedure  Training  Aids  which  present  textual  material 
In  an  Instructional  format  and  direct  students  how  to  use  the  material. 

Dr.  Friedman  presented  WANDAH  (Writing  Aid  AND  Author's  Helper),  an 
Intelligent  Instructional  system  designed  to  assist  writers  In  all  phases 
of  writing  Including  planning  and  organizing  Ideas,  transcribing  Ideas  Into 
print,  and  editing  and  revising.  Dr.  Glushko  described  a  system  called 
SOLID  (System  for  On-Line  Information  Development)  that  treats  document 
development  and  management  like  software  development  and  allows  for 
development  and  editing  of  multi  pie- author,  multi  pie- version  documents  with 
a  wide  range  of  edit  and  control  functions.  Ms.  Keenan  described  the 
WRITER'S  WORKBENCH  programs  designed  to  aid  In  the  editing  and  revision 
states  of  technical  documentation. 

Computer  Aided  Authoring  of  Procedure  Training  Aids  by  Richard  Braby, 
Training  Analysis  and  Evaluation  Group,  Naval  Training  Equipment  Center. 

Learning  to  operate  complex  equipment  Is  a  common  type  of  military 
training  activity.  Training  devices  and  operational  equipment  designated 
for  training  are  generally  provided  to  students  for  "hands-on*  practice. 
This  "hands-on"  training  Is  expensive  and  access  to  the  equipment  Is 
necessarily  limited.  To  make  the  best  use  of  this  time,  students  need 
preliminary  training.  For  example,  before  first  using  a  Cockpit  procedure 
Trainer,  the  student  Navy  pilot  studies  the  NATOPS  manual  and  other  printed 
materials  describing  a  specific  procedure.  However,  we  have  found  that 
this  kind  of  preliminary  training  Is  lacking  In  a  number  of  Important 
respects: 

o  Printed  materials  (e.g.,  workbooks  describing  procedures)  are  mainly 
narrat1ve--the  operating  Instructions  they  teach  are  heavily  visual  In 
nature. 

o  Class  Instruction  is  usually  In  a  group.  However,  learning  to 
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perform  procedures  Is  frequently  more  effective  when  practice  is 
sel f-paced. 

o  Students  are  frequently  unprepared  to  take  best  advantage  of  their 
limited  time  with  the  training  equipment.  When  properly  prepared, 
available  "hands-on"  time  produces  greater  training  gains. 

This  paper  describes  a  Procedure  Training  Aid  designed  to  overcome 
these  limitations.  It  presents  the  instructional  formats  for  the  material, 
describes  how  students  use  the  material  before  going  to  the  trainers,  and 
presents  the  results  of  a  field  trial.  Because  these  materials  are  fairly 
expensive  to  produce,  computer-aided  authoring  routines  have  been  created 
to  reduce  cost.  These  routines  are  described  along  with  sample  materials 
generated.  Finally,  recommendations  are  provided  on  how  best  to  use 
computer-generated  Procedure  Training  Aids. 

By  Incorporating  learning  principles,  our  Procedure  Training  Aids 
direct  the  student  In  efficient  practice  In  learning  to  perform  complex 
procedures.  Typically  Procedure  Training  Aids  have  four  types  of  pages: 

o  Presentation  Page  Presents  the  acts  necessary  to  perform  a  step 

In  the  procedure. 

o  Paraphrase  Page  Exercises  for  the  student  In  recalling  the 

content  of  the  previously  presented 
Presentation  Page. 

o  Road  Map  Page  With  guides  and  prompts,  provides  the  student 

with  an  exercise  In  which  he  talks  and  points 
his  way  through  a  group  of  steps  and  learns 
i.o  do  this  as  a  smooth  sequence  of  actions. 

o  Paper  Mock-up  Provides  the  student  with  an  extension  of  the 

Road  Map  exercise,  this  time  without  guides 
and  prompts. 

For  all  these  pages,  visual  job  task  Information  Is  presented 
graphically.  Words  are  used  only  for  those  messages  that  must  be 
transmitted  with  language.  Field  tests  have  shown  that  students  using 
Procedure  Training  Aids  perform  significantly  better  than  students  using 
more  traditional  types  of  materials. 

Procedure  Training  Aids  are  relatively  expensive  to  produce  using 
traditional  techniques.  Their  complex,  heavily  illustrated  pages  require 
considerable  time  and  expense  to  prepare.  In  addition,  a  larger  number  of 
pages  are  required  to  support  a  procedure  than  with  conventional  materials. 

Computer-aided  authoring  techniques  can  make  the  production  of  these 
materials  economical.  Two  generations  of  computer  routines  have  been 
developed  for  this  purpose,  and  a  third  Is  nearing  completion. 

These  routines  make  It  easy  and  fast  for  an  author  to  enter  procedure 
Information  Into  a  computer  data  base.  Only  plain  English  commands  are 
needed.  The  computer  then  automatically  formats  this  Information  and 
prints  camera-ready  copy  containing  the  formatted  text  and  boxes  for  the 
graphics  which  must  then  be  set  In  place  by  hand. 

The  first  generation  routine  Is  called  PLA  for  Computer  Automated  Page 
Layout  for  Text-Graphic  Materials  (Terrell,  1983).  It  Is  written  In  BASIC 
and  runs  on  the  WANG  2200  MVP  minicomputer.  The  basic  steps  In  using  this 
routing  are: 

o  The  author  enters  procedure  data  which  includes: 

-  steps  In  the  procedure 

-  dimensions  of  pictures  for  each  step 

-  text  for  labels 

-  picture-label  relationships 

-  page  headers  and  footers. 
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o  The  computer: 

-  selects  optimum  page  layouts 

-  draws  boxes  for  pictures  and  text;  lines  for  darts 

-  puts  text  It.  label  boxes 

-  types  headers  and  footers. 

o  The  author  then  completes  the  process: 

-  resolves  layout  problems  that  cannot  be  solved  by  the  PLA  routine 

-  places  pictures  In  designated  boxes 

-  places  adhesive-backed  darts  connecting  text  boxes  with  their 
points  of  action  on  the  graphics 

-  ensures  that  the  material  Is  camera  ready. 

A  second  generation  routing  called  MicroPLA  Is  similar  to  PLA  except 
that  It  runs  on  a  microcomputer,  the  Commodore  8032  and  and  contains  a 
large  number  of  default  values  that  reduce  the  time  required  to  enter  data 
Into  the  system. 


WANDAH  -  A  Computerized  Writer's  Aid  by  Morton  Friedman,  University  of 
California,  Los  Angeles. 

WANDAH  (Writing  Aid  ANO  Autnor's  Helper)  Is  a  new  generation 
Intelligent  Instructional  system  designed  to  assist  writers  In  all  phases 
of  writing  —  planning  and  organizing  Ideas,  transcribing  Ideas  Into  print, 
and  editing  and  revising.  The  design  of  WANDAH  draws  on  three  areas  of 
research:  first,  the  analysis  of  writing  as  a  cognitive  problem  solving 
activity;  second,  modern  composition  theory  and  research;  and  third, 
research  on  the  human-computer  Interface. 

WANDAH  has  three  major  components: 

1.  A  word  processor  designed  expressly  for  on-line  composing.  The 
word  processor  Includes  such  features  as  split  screen,  extensive 
on-line  keys,  easy-to-follow  menus,  and  special,  labelled  function 
keys. 

2.  A  set  of  prewriting  aids  incorporating  principles  of  modern 
composition  theory  that  help  writers  generate  Ideas  and  plan  their 
work. 

3.  A  set  of  aids  helping  writers  review  and  revise  their  work 
gramna tic ally,  stylistically,  and  thematically. 

These  aids  Include  spelling,  word  usage  and  punctuation  checkers,  a  style 
analyzer,  and  a  commenting  facility,  as  well  as  aids  that  create  an  ex-post 
facto  outline  of  the  work  and  that  examine  text  for  coherence  by  looking  at 
transition  phrases  and  pronouns. 

WANDAH  Is  written  In  UCSD  Pascal  and  runs  on  the  IBM-PC  and  similar 
machines. 


A  Software/Docuwentatlon  Development  Environment  Built  from  the  UNIX* 
Toolkit  by  Robert  J.  Glushko,  ATST  Bell  Laboratories 

Mismatches  In  technology,  media,  and  organizations  In  the  development 

*  UNIX  is  a  trademark  of  Bell  Laboratories.  My  comments  refer  to  the 
version  known  as  System  V  distributed  by  Western  Electric. 
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of  software  and  software  documentation  often  leave  the  documentation  late, 
incomplete,  and  poorly  coordinated  with  the  software.  One  solution  to  this 
problem  is  to  treat  software  documentation  like  software.  SOLID,  the 
System  for  On-Line  Information  Development,  builds  on  the  popular  UNIX 
operating  system  and  exploits  the  Idea  that  all  Information  that  can  be 
stored  on  a  computer  shares  a  common  life  cycle  and  can  be  created, 
managed,  and  delivered  with  the  same  methods  and  commands.  SOLID,  together 
with  the  UNIX  system,  solves  ten  key  problems  of  text  development  and 
management  that  can  be  faced  by  any  project. 

SOLID  Is  a  "Integrated"  programming  and  documentation  environment 
built  from  the  UNIX  system's  tools.  It  Implements  the  concepts  of  Source, 
where  all  the  source  for  programs  and  documents  are  kept,  Product,  where’ 
all  the  executable  programs  and  formatted  documents  reside,  and  generation 
Procedures,  which  translate  all  types  of  Source  Into  all  forms  of  Product. 
These  concepts  are  Implemented  as  UNIX  command  language  procedures  that 
combine  existing  tools.  SOLID  also  provides  Skeletons,  the  most  common 
outline  form  of  each  type  of  Source.  Developers- writing  a  new  module  of 
code  or  documentation  do  not  start  with  an  empty  file.  Instead,  they  are 
given  an  outline  which  reduces  drudgery  and  painlessly  enforces  some 
standards  of  style  and  content. 

SOLID  Is  notable  for  what  It  does  and  what  It  doesn't  do.  On  the 
"does"  side,  It  provides  a  single  environment  that  Is  being  used  to  develop 
all  documentation  for  computer  systems,  all  programs,  and  mixtures  of 
both.  It  comes  with  a  default  environment  that  Is  occasionally  used 
unmodified. 

On  the  other  side.  It  is  not  a  monolith  that  restricts  the  user  to 
those  options  that  are  built  In.  Since  all  SOLID  source  is  under  SOLID 
control,  the  user  can  change  or  extend  any  of  SOLID's  capabilities.  Some 
portions.  In  particular  Generation  Procedures,  are  Intended  to  be  extended 
to  support  new  Source  and  Product  types.  It  provides  no  editor;  people  use 
any  that  they  like.  It  doesn't  enforce  any  particular  programming  or 
documentation  methodology,  although  Skeletons  have  proved  useful  In 
encouraging  the  methodologies  and  styles  a  group  prefers. 

Although  released  only  recently  for  Internal  use,  SOLID  Is  now  used  by 
a  dozen  organizations  at  Bell  Laboratories.  Some  use  It  for  their 
documentation  only,  others  for  their  entire  product.  Because  It  Is  based 
on  observing  the  ways  people  have  customized  UNIX  to  suit  their  needs,  It 
seems  to  match  the  way  many  groups  do  business.  Because  It  Is  designed  to 
be  modifiable  and  Is  supported  by  its  own  mechanisms,  people  easily  extend 
and  customize  it  to  suit  their  own  needs.  Because  it  stands  firmly  on 
standard  UNIX,  as  UNIX  becomes  available  on  new  or  different  hardware, 

SOLID  Is  also.  Finally,  it  preserves  the  tool  orientation  of  UNIX.  New 
capabilities  added  to  UNIX  are  Immediately  available  to  SOLID  users. 


The  UNIX  WRITER'S  WORKBENCH  Software:  Computer  Aids  for  Text  Analysis  by 

Stacey  A.  Keenan,  AT&T  Bell  Laboratories . 

The  UNIX*  WRITER'S  WORKBENCH**  Software,  was  developed  at  AT&T  Bell 
Laboratories  to  aid  the  editing  and  revision  stages  of  technical  , 


UNfX  Is  a  trademark  of  AT&T  Bell  Laboratories. 

**  WRITER'S  WORKBENCH  Is  a  trademark  of  AT&T  Technologies. 
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documentation.  In  designing  the  WRITER'S  WORKBENCH  system,  we  followed  six 
design  principles  which  can  apply  to  many  kinds  of  software  systems.  We 
felt  the  programs  should  be: 

-  rational —programs  are  based  on  expert  knowledge  about  what  makes 
documents  difficult  to  understand. 

-  diverse— programs  reflect  the  complexity  of  communication  by 
reporting  only  aspects  of  document  style.  One  score  does  not  describe 
the  quality  of  a  document. 

-evaluative  relative  to  standards- -documents  are  judged  according  to 
their  Intended  audience. 

-  modifiable— users  can  select  standards,  build  new  standards,  modify 
dictionaries,  and  request  short  or  long  output. 

-  specific-programs  tell  what  problems  a  text  has,  and  shows  where 
those  problems  occur. 

-  Informative— programs  provide  Information,  but  do  not  make  rigid 
decisions,  or  make  changes  to  text. 

The  WRITER'S  WORKBENCH  programs  analyze  prose  documents  stored  In  the 
computer  and  suggest  Improvements  to  the  writing.  The  user  evaluates  the 
suggestions  and  decides  vtoat  to  change.  The  programs  can  be  divided  Into 
four  major  areas:  1)  proofreading,  2)  stylistic  analysis,  3)  on-line 
Information  about  English  usage  and  about  the  WRITER'S  WORKBENCH  programs, 
and  4)  utility  programs,  for  modifying  other  programs.  The  programs  are 
designed  to  provide  as  much  or  as  little  Information  as  the  user  requires. 

The  WRITER'S  WORKBENCH  system  has  been  widely  used  at  AT&T  Bell 
Laboratories  for  over  4  years.  Freshman  composition  students  at  Colorado 
State  University  and  their  teachers  are  enthusiastic  about  using  the 
programs.  Use  of  the  WRITER'S  WORKBENCH  system  speeds  editing,  heightens 
authors'  awareness  of  writing  style,  and  frees  the  author  (or  teacher)  to 
concentrate  on  more  substantive  Issues  of  document  content. 


Session  V:  Application  of  Artificial  Intelligence  to  Autaaated  Authoring 

The  papers  In  this  session  explored  future  directions  of  author  aiding 
systems  with  a  particular  emphasis  on  the  potential  contribution  of 
artificial  Intelligence  to  these  systems.  Dr.  Jensen  described  research 
and  development  efforts  to  achieve  an  expert  training  system  technology 
that  Is  approaching  applied  artificial  intelligence  methodology  in  training 
systems.  Dr.  Stacy  presented  a  schema  for  Intelligent  computer-assisted 
Instruction  on  troubleshooting.  Dr.  Spiro  drew  the  distinction  between 
well-  and  Ill-structured  domains  of  knowledge  representation  and 
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application  and  described  an  instructional  approach,  Experience- 
Consolidation  Systems,  that  relies  upon  the  human  visual -perceptual 
processing  system  to  acquire  knowledge  in  11 1 -structured  domains.  Dr. 
Gibbons  concluded  the  session  by  documenting  the  major  costs  residing  In 
the  development  and  delivery  of  computer-based  training,  demonstrating  the 
economic  effects  of  efficient  systems,  and  enumerating  the  features  that 
future  authoring  systems  must  have  to  be  efficient. 


The  Boeing  Company  Training  Equipment  and  Simulator  Story  by  J.  Marty 
Jensen,  Boeing  Aerospace  Company. 

Over  the  past  6  years,  the  Boeing  Training  Systems  Organization  has 
expended  research  and  development  effort  In  achieving  an  expert  training 
system  technology  that  Is  approaching  applied  artificial  Intelligence 
methodology  In  training  systems. 

Our  philosophy  Is  based  on  the  principle  of  "The  End  Depends  on  the 
Beginning1'  for  the  design  of  our  Database  Management  System  Controller 
Software  Executive.  The  computer  driven  training  database  executive  Is 
designed  with  Intelligent  friendly  logic  that  helps  student  learning  and 
aiding  to  task  completion  through  the  delivery  of  the  database  elements. 

The  database  contains  the  elements  necessary  to  transfer  knowledge  to  users 
and  to  guide  practice  of  operating  procedures.  The  database  Is  Input  by 
the  user  and  establishes  the  media  for  the  system  executive.  A  primary 
requirement  for  the  database  Involves  detailed  planning  and  front  end 
analysis  to  establish  the  Intelligence  for  the  Expert  System.  The  logic 
for  the  Expert  System  Executive  provides  Instructions  help/aldlng  and 
remediation  to  the  user  during  the  execution  of  the  learning  process.  The 
executive  also  monitors  and  maintains  records  of  Individual  performance. 

The  Expert  System  philosophy  has  been  applied  In  several  training 
system  applications.  User  friendly  software  Is  provided  for  easy  Input  and 
control  of  the  database  and  for  steering  the  Expert  System  Executive  for 
delivery  of  the  database. 

Evidence  of  the  effectiveness  of  training  developed  using  this  system 
emerged  from  an  evaluation  of  one  application  -  the  Maintenance 
Institutional  Trainer  (MIT).  Two  formal  classes  trained  on  the  MIT.  Its 
training  effectiveness  was  far  in  excess  of  the  users  original 
expectations.  The  U.S.  Army  training  community  estimated  that  an 
additional  4  to  6  weeks  of  training  time  would  have  to  be  added  to  the 
course  of  Instruction  If  the  MIT  were  not  available.  An  Important  added 
benefit  was  the  reduced  need  to  train  on  tactical  hardware.  With  the  MIT 
available,  a  class  size  of  IS  students  only  required  the  use  of  two 
tactical  systems  while  without  the  MIT  a  minimum  of  six  tactical  systems 
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would  have  oeen  needed. 

The  effectiveness  of  the  trainer  provided  other  benefits  also.  The 
CA1  feature  promoted  self-paced  training  capability.  The  management 
feature  provided  course  administrative  support.  Both  features  resulted  in 
reduced  requirements  for  Instructor  and  overhead  personnel  support. 

In  summary,  the  Maintenance  Institutional  Trainer  was  an  extremely 
effective,  state  of  the  art  training  device  that  should  be  further 
developed  for  training  on  additional  weapons  systems. 

During  the  past  year,  Boeing  has  assembled  a  laboratory  trainer 
generic  development  station  based  on  lessons  learned  from  previous  trainer 
designs  for  the  purpose  of  experimental  application  of  technologies  In 
hardware,  software,  courseware  authoring,  and  training  records.  The 
station  Is  used  to  test  and  demonstrate  various  trainer  Interfaces  for 
Interactive  student  participation.  It  Incorporates  2D  and  3D  equipment  for 
testing  methods  and  principles  for  delivery  of  training  sequences.  This 
allows  front  end  application  of  courseware  techniques  prior  to  commltlng  to 
a  specific  trainer  baseline  configuration.  A  VAX  11/750  is  the  host 
processor  which  contains  a  trainer  executive  control  system  programmed  In 
FORTRAN.  It  Interfaces  with  a  smart  I/O  system  that  monitors  switch  Inputs 
and  controls  outputs  to  3D  simulated  test  equipment  displays,  analog  and 
digital  devices.  Standard  terminals,  a  graphic  display,  a  record  and 
playback  video  disc  and  various  Interactive  Input  devices  are  also  Included 
for  2D  simulation.  A  courseware  authoring  language  called  ALPHA  Is  the 
user  Interface  to  Integrate  these  devices  for  delivery  of  training 
scenarios.  The  authoring  system  Is  a  computer  aided  process  which  allows 
courses  to  be  entered  Into  the  computer  database  for  both  the  3D  and  2D 
devices.  This  system  Is  now  able  to  demonstrated  and  available  to  evaluate 
user  requirements  and  future  research  application. 


A  Schema  for  ICAI  on  Troubleshooting  by  E.  Webb  Stacy,  Jr.,  Scientific 
Systems,  Inc. 

The  purpose  of  this  talk  was  to  sketch  features  of  a  system  that  can 
provide  Intelligent  computer-aided  Instruction  on  troubleshooting.  In 
keeping  with  the  theme  of  automated  authoring  aids,  the  system  will  ease 
the  process  of  developing  training  for  troubleshooting  a  system  to  the 
extent  that  generic  troubleshooting  knowledge  applies  to  that  system. 

We  envision  the  system  to  have  three  major  components:  a  knowledge 
base,  a  model  of  the  student,  and  an  Instructional  engine.  The  knowledge 
base  contains  the  Information  to  be  taught,  the  student  model  represents 
the  student's  status  with  respect  to  the  knowledge  base,  and  the 
Instructional  engine  provides  Instruction  with  the  goal  of  moving  the 
student  model  (and  therefore  the  student)  In  the  direction  of  the  knowledge 
base  (and  therefore  the  expert.) 

It  Is  necessary  that  the  knowledge  representation  strategy  In  the 
knowledge  base  have  certain  features  isomorphic  to  the  way  the  expert 
actually  represents  knowledge.  An  'ttempt  tc  train  medical  residents  using 
the  rules  In  MYC IN,  a  medical  diagnostic  expert  system,  showed  that  the 
rules  were  not  particularly  comprehensible  to  humans;  a  reformulation  of 
the  rules  In  a  form  more  natural  to  humans  (NEOMYCIN)  resulted  In  no 
decrement  In  performance  as  an  expert  system  but  much  more  effective 
training.  Hence  It  Is  Important  to  understand  the  nature  of  expert 
knowledge. 
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Expert  troubleshooters  can  minimize  total  troubleshooting  time  by 
considering  simultaneously  the  time  costs  of  tests,  the  informativeness  of 
tests,  and  the  relative  reliability  of  system  components.  They  have 
pragmatic,  but  not  necessarily  detailed,  knowledge  of  system  and  subsystem 
functions,  usually  have  multiple  representations  of  the  systems  (e.g., 
logical  and  physical),  and  have  a  large  vocabulary  of  recognizable  symptom 
patterns.  This  knowledge  will  therefore  be  a  necessry  component  of  the 
knowledge  base. 

The  student's  knowledge  may  differ  from  the  expert's  in  that  the 
student  may  have  missing  knowledge.  For  instance,  student  troubleshooters 
have  been  shown  to  omit  one,  two,  or  all  of  the  time  the  cost  of  tests,  the 
Informativeness  of  tests,  or  component  reliability.  The  student  may  also 
have  extra  (Inaccurate)  knowledge,  which  may  take  the  form  of 
overgeneralization,  overdifferentiation,  or  misconceptions.  Finally,  the 
student's  knowledge  Is  slower  and  more  effortful  than  the  expert's. 

The  Instructional  engine  will  deal  with  missing  knowledge  via 
conventional  representation  methods;  with  extra  knowledge  via  debugging  of 
student  knowledge  using  well-chosen  counterexamples;  and  with  slow 
performance  via  practice  with  feedback  In  a  simulated  environment. 

Clearly,  there  will  be  features  of  an  ICA1  system  for  troubleshooting 
a  specific  system  that  are  unique  to  that  system.  We  do  not  believe  that 
troubleshooting  knowledge  applies  mutatls  mutandis  for  all  systems.  We  do 
believe  that  there  Is  much  commonality  In  the  way- expert  troubleshooters 
think  about  their  systems,  and  In  effective  strategies  for  training 
troubleshooting  skills.  By  arranging  an  ICAI  system  as  described  here, 
with  careful  attention  to  facilitating  modification  of  Its  system-specific 
aspects,  development  effort  for  training  the  troubleshooting  of  a  new 
system  will  be  greatly  reduced. 

Automated  Author-Aids  for  Learning  fro*  Cases  In  111 -Structured  Knowledge 
Domains  by  Rand  J.  Spiro,  University  of  Illinois  at  Urbana. 

The  distinction  was  drawn  between  well-  and  Ill-structured  domains  of 
knowledge  representation  and  application.  In  Ill-structured  domains  there 
1$  no  suitable  set  of  defining  characteristics  for  category  membership, 
proper  hierarchy  assumptions  are  violated  across  Individual  cases,  and 
category  prototypes  tend  to  be  frequently  misleading  —  in  short,  general 
principles  are  highly  misleading  by  themselves.  It  was  argued  that  most 
domains  are  more  ill-structured  that  Is  typically  acknowledged.  In  such 
situations  learning  Is  experience-intensive,  requiring  exposure  to  large 
numbers  of  cases  (examples,  events,  Instances,  situations,  etc.),  with  rich 
and  highly  Interconnected  memory  representation  of  the  patterns  of 
attribute-configurations  from  the  Individual  cases. 

These  requirements  pose  instructional  problems  because  of  the  demands 
they  place  on  exposure  time,  memory  load,  and  sorting  out  the  Inefficacious 
natural  arrangement  of  cases.  An  approach  that  overcomes  each  of  these 
problems  was  presented:  Experience-Consolidation  Systems  (ECSs).  ECSs 
present  In  a  mnemonlcally  convenient  form  much  of  tne  Information  that 
normally  Is  acquired  from  field  experience,  except  In  hours  Instead  of 
years.  They  rely  on  the  human  visual -perceptual  processing  system's  Innate 
ability  to  simultaneously  represent  complex  and  Irregular 
Interrelationships,  something  for  which  the  more  verbal  and  analytic 
processing  system  Is  poorly  suited.  Integrated  visual  analogues  of  a 
nearly  exhaustive  set  of  perspectives  for  oescriblng  an  individual  case  are 
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presented  and  re-oresented  In  varying  contexts  by  the  computer  using  a 
unique  arrangement  algorithm  that  "cri ss-crosses"  the  set  of  cases  in  such 
a  way  as  to  facilitate  the  noticing  of  conflgural  attributes  of  resemblance 
within  the  outcome  "families"  that  need  to  be  distinguished.  Thus  the 
performance  process  engendered  Is  one  of  perceptual  recognition.  The 
principle  that  enables  correct  recognition  Is  very  similar  to  that  of  real 
gene-pool  Inheritance  and  family  resemblance. 

Among  many  other  uses,  EC$s  have  great  potential  value  In  military 
settings  given  the  poor  literacy  skills  of  many  recruits  and  the  prevalence 
of  experience-intensive,  Ill-structured  military  situations  that  make  the 
development  of  systems  to  consolidate  the  acquisition  of  case  experience 
Imperative  If  personnel  are  to  efficiently  learn  how  to  make  accurate 
judgments  and  decisions  when  rules  presented  In  formal  Instructional 
settings  do  not  generalize  sufficiently. 


Economics  of  Author-Aiding  by  Andrew  S.  Gibbons,  WICAT  Systems,  Inc. 

The  form  and  extent  of  future  author-aiding  systems  will  depend  on  how 
well  they  make  authoring  less  costly  or  more  productive. 

As  the  military  and  business  communities  have  Increased  their 
commitment  to  computer-based  training  (CST)  by  authoring  thousands  of  hours 
of  Instruction,  It  Is  becoming  clear  that  the  major  costs  of  CBT  reside  In 
development  and  delivery  and  not  In  hardware.  This  report  demonstrates  the 
economic  effect  of  efficient  systems  and  enumerates  features  that  efficient 
authoring  and  author-aiding  systems  must  have. 


Small  Group  Discussions  and  Conference  Summary 

Sessions  VI,  VII,  and  VIII  were  devoted  to  panel  discussions  of  Issues 
emerging  from  the  presentations.  Participants  were  assigned  to  panels  to 
Insure  a  mix  of  Individuals  representing  the  military,  private,  and 
academic  sector.  Each  of  the  civilian  consultants  served  as  the 
facilitator  for  one  of  the  panels.  The  questions  which  structured  these 
discussions  and  presentations  are  Included  In  the  Appendix. 

Dr.  Dexter  Fletcher  of  the  University  of  Oregon  served  as  the 
conference  summarizer.  His  comments  were  based  on  the  presentations  and 
panel  discussions  that  were  held  over  the  two  days.  His  approach  was  to 
Identify  anomalies  that  he  had  noted  during  the  proceedings,  as  a  way  of 
highlighting  the  Issues  which  must  be  addressed  in  the  further  development 
and  use  of  author  aiding  systems.  These  anomalies  Include: 
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1.  The  technical  adequacy  of  ISP.  While  specific  concerns  were 
raised  about  the  ISO  model,  there  was  general  agreement  that  some  type  of 
logical,  structured  analysis  and  development  procedure  such  as  ISO  is 
necessary  to  design  any  Instruction,  in  fact,  there  Is  some  evidence  that 
applying  ISD  procedures  to  courses  before  they  are  presented  on  computers 
may  actually  account  for  their  success  rather  than  the  computer  technology 
Itself  (see,  for  example,  Orlansky  and  String's  review  of  computer  based 
Instruction  In  the  military).  While  there  Is  evidence  to  believe  that  ISD 
has  value  as  a  technical  approach,  certain  limitations  and  technical 
difficulties  were  consistently  noted  by  conference  participants.  For 
example,  while  the  procedural  steps  In  USD  are  logically  organized,  some  of 
the  blocks  within  the  ISD  process  require  complex  and  difficult  decisions, 
particularly  the  Initial  blocks  of  front-end  analysis  activities, 

Involving  task  analysis  and  Instructional  objectives  definition.  These 
analysis  activities  are  often  not  straightforward,  particularly  for  complex 
congltlve  job-tasks;  further  complicating  the  analytic  work  Is  this 
practical  circumstance  that  access  to  subject  matter  experts  Is  usually 
quite  limited.  Furthermore,  It  Is  often  hard  for  subject  matter  experts  to 
articulate  the  key  features  of  a  task  or  to  describe  standards  for 
determining  the  adequacy  of  task  performance.  This  Is  especially  true  for 
cognitive  tasks  requiring  higher  order  processing  of  Information  and 
decision  making.  Greater  attention  needs  to  be  directed  toward 
articulation  of  how  to  perform  ISD  prcedures  more  effectively. 

2.  The  implementation  of  ISD.  One  thread  that  ran  throughout  the 
comments  of  many  participants  was  the  administrative  and  management 
problems  associated  with  the  implementation  of  ISD  which  have  Impeded  its 
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success,  These  problems  Include  Inadequate  training  of  ISD  managers  and 
practitioners  as  well  as  Inadequate  resources  (Including  time  and 
expertise)  applied  to  ISO  procedures.  In  addition,  corrmunl cation  with  job 
managers  In  the  field,  as  well  as  scheduling  constraints  and  availability 
of  subject  matter  experts  contribute  further  to  problems  In  the  effective 
Implementation  of  the  ISD  model. 

3.  Authoring  by  Individuals  with  team  review.  While  Individuals 
typically  write  Initial  drafts  (e.g.,  a  task  analysis,  training  objectives, 
lesson  plans,  a  test),  their  work  is  generally  reviewed  extensively  both  by 
other  Individuals  and  by  committees.  The  consensus  view  appeared  to  be 
that,  not  only  would  Intelligent  computer  aiding  contribute  to  the 
productivity  and  effectiveness  of  Individual  authors,  but  that  the  review 
process  could  be  made  shorter  and  more  efficient  with  an  aiding  system  that 
Incorporated  such  features  as  E  mall  and  word  processing. 

4.  The  role  of  cognitive  science.  There  Is  a  great  deal  of  talk 
about  the  Importance  of  Incorporating  principles  of  cognitive  science  Into 
the  ISD  process  or  of  basing  our  notions  about  Instructional  development  on 
Ideas  emerging  from  cognitive  science  and  the  study  of  human  Information 
processing.  However,  It  Is  very  difficult  for  Instructional  developers  and 
users  of  ISD  to  articulate  exactly  how  specific  principles  from  cognitive 
science  were  or  could  be  Incorporated  Into  their  Instructional  development 
activities.  More  attention  needs  to  devoted  to  this  Issue. 

5.  The  range  of  training  problems.  Sources  of  training  problems  not 
only  Include  problems  stemming  from  the  ISD  process  Itself,  but  also 
Include  problems  arising  from  a  broad  range  of  other  sources  specifically 
Including  selection  and  placement  problems  (e.g.,  trainees  with  Inadequate 
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entering  skills  and  knowledge),  resource  allocation  problems  (e.g., 
Insufficient  time  for  accomplishing  training  objectives),  and  management 
problems  within  the  employing  organization  rather  than  the  training 
organization  (e.g.,  time  lag  between  training  and  job  assignment  resulting 
In  skill  decay,  mismatch  between  training  and  job  requirements).  Effort 
should  be  directed  to  examining  the  nature  of  "training"  problems  beyond 
the  of  1SD  narrowly  conceived  and  alternative  solution  strategies 
considered. 

6.  The  need  for  a  systems  approach.  There  Is  a  need  for  ISD  to  be 
Incorporated  Into  a  broader  systems  view  so  that  ISD  does  not  start  with 
the  assumption  that  training  Is  the  appropriate  solution  to  a  given  problem 
or,  equivalently,  so  that  ISD  does  not  stop  with  training  delivery.  A  true 
systems  approach  would  consider  the  full  range  of  personnel  and 
organizational  management,  not  just  the  contributions  of  training  and  would 
provide  a  mechanism  for  helping  to  decide  If  the  cost-effective  solution  is 
to  seek  better  training.  In  a  given  situation  It  might  be,  for  example, 
that  the  cost-effective  approach  Is  to  seek  better  personnel  selection,  or 
better  Job  classification,  or  better  job  design,  or  better  equipment 
design,  or  better  job  aids,  or  better  supply  schedules,  etc.  Thus,  ISD 
ought  to  be  viewed  In  a  context  broader  than  training. 

7.  The  capabilities  of  extant  hardware  and  software.  There  Is  a 
common  perception  that  efforts  In  training  lag  behind  the  computer 
revolution  In  software  and  hardware.  However,  there  are  ISD  needs  that  far 
outstrip  the  capabilities  of  current  technology.  This  situation  Is 
particularly  true  In  the  field  of  artificial  Intelligence  and  has  direct 
Implications  for  the  development  of  an  appropriately  powerful  AI  system  for 
ISD. 
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8.  The  role  of  artificial  Intelligence.  While  artificial 
Intelligence  (Al)  Is  often  viewed  as  having  great  potential  for  application 
to  the  ISD  process,  there  may  be  limitations.  If  ISD  Is  conceived  as 
common  sense  made  systematic,  then  it  may  be  anomalous  to  consider  AI  as  a 
promising  approach  in  ISD  since  the  common  lament  among  AI  workers  Is  their 
Inability  to  capture  "common  sense*.  It  Is  also  Important  to  consider  the 
costs  of  adding  AI  to  the  ISD  process.  That  Is,  It  will  be  Important  to 
estimate  whether  the  costs  to  be  incurred  are  compensated  for  by  Increased 
instructional  effectiveness  or  Increased  productivity  among  Instructional 
developers. 


Future  Research 

What  future  directions  should  research  and  development  efforts  In  the 
area  of  automated  author  aiding  systems  take?  A  view  frequently  expressed 
by  conference  participants  was  that,  on  the  one  hand,  It  was  too  early  to 
plan  for  a  fully  automated  Intelligent  authoring  system.  However  on  the 
other  hand,  many  computer-based  procedures  could  be  put  to  good  use.  An 
Instructional  Designer's/Instructor's  Workbench  serves  as  a  good  example. 
Such  a  system  would  eventually  Include  an  expert  system  for  Instructional 
design  principles,  as  well  as  systems  for  word  processing,  guidance  In 
Instructional  design,  and  temp late -based  author  aiding. 

To  facilitate  rapid  application  of  automated  author  aiding 
tools,  a  number  of  parameters  were  judged  as  appropriate  constraints  for 
Initial  development  activity.  These  were: 

o  a  focus  on  Individual  worker  rather  than  group  training; 
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o  use  of  a  technical  "hard"  skill  domain  such  as  operator  or 
maintenance  training  rather  than  a  "soft"  skill  domain  such  as 
leadership; 

o  requirement  for  adequate  rather  than  optimal  Instruction; 
o  In-school  Instruction  rather  than  unit  training; 
o  off-the-shelf  hardware  rather  than  hardware  development; 
o  examination  of  existing  systems  and  Identification  of  needed 
enhancements; 

o  consideration  of  artificial  Intelligence  techniques  particularly 
for  the  knowledge  base  of  instructional  principles; 
o  rapid  development  with  formative  evaluation  and  revision  rather 
than  an  extended  development  cycle. 
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APPENDIX 


A-l 


Structuring  Questions  for  Small  Group  Discussions 
(Sessions  VI ,  VII ,  VIII) 


1.  For  what  functions,  e.g.,  analysis,  instructional  design,  are  auto¬ 
mated  authoring  aid  technologies  likely  to  have  the  largest  short¬ 
term  (3-5  years)  impact? 

2.  What  functions  (becaus'  ■  <f  their  success)  no  longer  require  additional 
R&D  effort? 

3.  What  functions  should  be  discarded  temporarily  (because  of  lack  of 
knowledge)  or  permanently  (because  of  lack  of  progress  or  potential 

impact )? 

4.  What  functions  or  areas  suggest  the  need  for  relatively  large  investment 
for  a  probable  short-term  (3-5  years)  impact? 

5.  What  should  be  criteria  for  the  assessment  of  Internal  features  and  ex¬ 
ternal  effects  of  AAAS? 

6.  For  what  learning  tasks  are  AAAS  best  suited? 

7.  What  new  training  requirements  for  AAA  implementation  should  be  de¬ 
veloped? 

8.  Mow  extensible  are  extant  designs  for  AAAS  to  a  wide  range  of  tasks 
and  delivery  conditions?  What  can  be  improved  rapidly? 
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Courseware  bottleneck  in  CAI  systems 
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Objective:  State  of  Art  Assessment 
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AI  EXPERT  SYSTEM  FOR  MEDIA  SELECTION 


Approach:  State  of  the  Art  Assessment 
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Figure  2-2.  A  model  for  a  systems  approach  to  training. 
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